[BOLT] Match functions with pseudo probes #100446

shawbyoung · 2024-07-24T18:28:19Z

Extended function and block pseudo probe stale matching

Follow-up to #99891.

This change brings flexible profile recovery using pseudo probes:

using multiple profiles for one binary function, e.g. if the binary
function inlines profile functions,
using parts of one profile across multiple binary functions, e.g. if
the profile has more aggressive inlining compared to the input binary.

Prerequisites

YAML profile must be created with --profile-write-pseudo-probes
perf2bolt flag.
Pseudo probe matching is enabled with
--stale-matching-with-pseudo-probes which augments hash-based
matching + profile inference enabled with --infer-stale-profile
(required).
The binary must contain pseudo probes enabled with
-fpseudo-probe-for-profiling Clang option.

Performance testing

Using two LTO+CSSPGO builds of Clang ([Clang][CMake] Add CSSPGO support to LLVM_BUILD_INSTRUMENTED #79942) from the same source
with different profiles causing different inlining decisions,
Clang-BOLT testing harness https://github.com/aaupov/llvm-devmtg-2022
2S Intel SKX Xeon 6138 with 40C/80T and 256GB RAM, using 20C/40T for
build,
Bootstrapped build of Clang, wall time:
- no BOLT: 301.87 +- 1.44 seconds, 0.91 IPC
- stale profile: 295.242 +- 0.683 seconds, 0.93 IPC
- hash matching: 289.035 +- 0.656 seconds, 0.95 IPC
- probe matching: 283.558 +- 0.927 seconds, 0.96 IPC
- fresh profile: 275.131 +- 0.444 seconds, 0.99 IPC

Depends on:

Co-authored-by: aaupov@fb.com

Created using spr 1.3.4

To be used for pseudo probe function matching (#100446). Test Plan: updated pseudoprobe-decoding-inline.test Pull Request: #107137

…BF, determine best matching binary inline subtrees, find corresponding binary functions, attach the profile to binary function without a profile and most probes Created using spr 1.3.4

Created using spr 1.3.4

github-actions · 2024-11-08T21:15:09Z

✅ With the latest revision this PR passed the C/C++ code formatter.

Created using spr 1.3.4

Match inline trees first between profile and the binary: by GUID, checksum, parent, and inline site for inlined functions. Map profile probes to binary probes via matched inline tree nodes. Each binary probe has an associated binary basic block. If all probes from one profile basic block map to the same binary basic block, it’s an exact match, otherwise the block is determined by majority vote and reported as loose match. Pseudo probe matching happens between exact hash matching and call/loose matching. Introduce ProbeMatchSpec - a mechanism to match probes belonging to another binary function. For example, given functions foo and bar: ``` void foo() { bar(); } ``` profiled binary: bar is not inlined => have top-level function bar new binary where the profile is applied to: bar is inlined into foo. Currently, BOLT does 1:1 matching between profile functions and binary functions based on the name. #100446 will extend this to N:M where multiple profiles can be matched to one binary function (as in the example above where binary function foo would use profiles for foo and bar), and one profile can be matched to multiple binary functions (e.g. if bar was inlined into multiple functions). In this diff, ProbeMatchSpecs would only have one BinaryFunctionProfile (existing name-based matching). Test Plan: Added match-blocks-with-pseudo-probes.test Performance test: - Setup: - Baseline no-BOLT: Clang with pseudo probes, ThinLTO + CSSPGO (#79942) - BOLT fresh: BOLTed Clang using fresh profile, - BOLT stale (hash): BOLTed Clang using stale profile (collected on Clang 10K commits back), `-infer-stale-profile` (hash+call block matching) - BOLT stale (+probe): BOLTed Clang using stale profile, `-infer-stale-profile` with `-stale-matching-with-pseudo-probes` (hash+call+pseudo probe block matching) - 2S Intel SKX Xeon 6138 with 40C/80T and 256GB RAM, using 20C/40T for build, - BOLT profiles are collected on Clang compiling large preprocessed C++ file. - Benchmark: building Clang (average of 5 runs), see driver in aaupov/llvm-devmtg-2022 - Results, wall time, lower is better: - Baseline no-BOLT: 429.52 +- 2.61s, - BOLT stale (hash): 413.21 +- 2.19s, - BOLT stale (+probe): 409.69 +- 1.41s, - BOLT fresh: 384.50 +- 1.80s. --------- Co-authored-by: Amir Ayupov <aaupov@fb.com>

Created using spr 1.3.4

Pseudo probe matching (#100446) needs callee information for call probes. Embed call probe information (probe id, inline tree node, indirect flag) into CallSiteInfo. As a consequence: - Remove call probes from PseudoProbeInfo to avoid duplication, making it only contain block probes. - Probe grouping across inline tree nodes becomes more potent + allows to unambiguously elide block id 1 (common case). Block mask (blx) encoding becomes a low-ROI optimization and will be replaced by a more compact encoding leveraging simplified PseudoProbeInfo in #166680. The size increase is ~3% for an XL profile (461->475MB). Compact block probe encoding shrinks it by ~6%. Test Plan: updated pseudoprobe-decoding-{inline,noinline}.test

Pseudo probe matching (#100446) needs callee information for call probes. Embed call probe information (probe id, inline tree node, indirect flag) into CallSiteInfo. As a consequence: - Remove call probes from PseudoProbeInfo to avoid duplication, making it only contain block probes. - Probe grouping across inline tree nodes becomes more potent + allows to unambiguously elide block id 1 (common case). Block mask (blx) encoding becomes a low-ROI optimization and will be replaced by a more compact encoding leveraging simplified PseudoProbeInfo in #166680. The size increase is ~3% for an XL profile (461->475MB). Compact block probe encoding shrinks it by ~6%. Test Plan: updated pseudoprobe-decoding-{inline,noinline}.test Reviewers: paschalis-mpeis, ayermolo, yota9, yozhu, rafaelauler, maksfb Reviewed By: rafaelauler Pull Request: #165490

wlei-llvm · 2025-11-17T19:50:58Z

bolt/lib/Profile/YAMLProfileReader.cpp

+    for (auto &[ParentId, InlineSiteCSI] : ParentToCSI) {
+      for (auto &[InlineSite, CSI] : InlineSiteCSI) {
+        auto [CalleeGUID, CallSite] = InlineSite;
+        errs() << ParentId << "@" << CallSite << "->"


errs() --> dbgs()

wlei-llvm · 2025-11-17T19:54:40Z

bolt/lib/Profile/YAMLProfileReader.cpp

+    ++ProbeMatchingStats.MismatchingRootGUID;
+    return 0;
+  }
+  {


why use the extra { }?

wlei-llvm · 2025-11-17T19:56:22Z

bolt/lib/Profile/YAMLProfileReader.cpp

+  {
+    uint64_t BinaryHash = Decoder.getFuncDescForGUID(Root.Guid)->FuncHash;
+    uint64_t ProfileHash = FuncNode.Hash;
+    if (BinaryHash != ProfileHash) {


This is due to the stale profile, right?

wlei-llvm · 2025-11-17T19:57:51Z

bolt/lib/Profile/YAMLProfileReader.cpp

+  ++ProbeMatchingStats.MatchedRoots;
+
+  uint64_t Matched = 1;
+  for (const auto &[Idx, ProfileNode] :


Could you add some comments for this for loop?

wlei-llvm · 2025-11-17T20:03:11Z

bolt/lib/Profile/YAMLProfileReader.cpp

+  return Root;
+}
+
+size_t YAMLProfileReader::matchInlineTreesImpl(


Could you add some more high-level descriptions(like more details for what each step does and for different cases) for this function? The function is complex, and it would help future readers.

Also do you think we have good test coverage for this function? it looks it all reuses the existing tests, no test for the new feature? Though I understand it probably needs a large binary/profile, if that's the case, i'm also fine as long as you have tested locally.

wlei-llvm · 2025-11-17T20:03:24Z

bolt/lib/Profile/YAMLProfileReader.cpp

+    BinaryFunction &BF, yaml::bolt::BinaryFunctionProfile &YamlBF,
+    const MCDecodedPseudoProbeInlineTree &Root, uint32_t RootIdx,
+    ArrayRef<yaml::bolt::InlineTreeNode> ProfileInlineTree,
+    MutableArrayRef<const MCDecodedPseudoProbeInlineTree *> Map, float Scale) {


Looks scale is unused

wlei-llvm · 2025-11-17T20:10:53Z

bolt/lib/Profile/YAMLProfileReader.cpp

    }
  }
+  if (opts::StaleMatchingWithPseudoProbes) {
+    SmallVector<StringRef, 0> Suffixes(


maksfb · 2025-11-18T04:24:07Z

bolt/lib/Profile/YAMLProfileReader.cpp

+    uint64_t BinaryHash = Decoder.getFuncDescForGUID(Root.Guid)->FuncHash;
+    uint64_t ProfileHash = FuncNode.Hash;


nit: constify.

maksfb · 2025-11-18T04:26:03Z

bolt/include/bolt/Profile/YAMLProfileReader.h

    ProfiledFunctions.emplace(&BF);
  }

+  /// Return a top-level binary inline tree node for a given \p BF


Suggested change

/// Return a top-level binary inline tree node for a given \p BF

/// Return a top-level binary inline tree node for a given \p BF.

maksfb · 2025-11-18T04:27:55Z

bolt/lib/Passes/BinaryPasses.cpp

        BC.Stats.ExactMatchedSampleCount, BC.Stats.StaleSampleCount);
    BC.outs() << format(
-        "BOLT-INFO: inference found an exact pseudo probe match for %.2f%% of "
+        "BOLT-INFO: inference found pseudo probe match for %.2f%% of "


Do we want to print this info if pseudo probes are not present?

maksfb · 2025-11-18T04:29:23Z

bolt/lib/Profile/StaleProfileMatching.cpp

-    HashFunction HashFunction, YAMLProfileReader::ProfileLookupMap &IdToYamlBF,
-    const BinaryFunction &BF,
-    const ArrayRef<YAMLProfileReader::ProbeMatchSpec> ProbeMatchSpecs);
+size_t matchWeights(BinaryContext &BC,


BinaryContext can be obtained from BinaryFunction.

maksfb · 2025-11-18T04:34:17Z

bolt/lib/Profile/StaleProfileMatching.cpp

    BlendedBlockHash BinHash = BlendedHashes[MatchedBlock->Index - 1];
-    LLVM_DEBUG(dbgs() << "Matched yaml block (bid = " << YamlBBIdx << ")"
-                      << " with hash " << Twine::utohexstr(YamlBB->Hash)
+    LLVM_DEBUG(dbgs() << "Matched yaml block (bid = " << YamlBB.Index << ")"


Suggested change

LLVM_DEBUG(dbgs() << "Matched yaml block (bid = " << YamlBB.Index << ")"

LLVM_DEBUG(dbgs() << "BOLT-DEBUG: matched yaml block (bid = " << YamlBB.Index << ")"

maksfb · 2025-11-18T04:48:35Z

bolt/lib/Profile/YAMLProfileReader.cpp

+      }
+      const BinaryFunctionProfile *Callee = IdToYamLBF.lookup(CSI.DestId);
+      if (!Callee) {
+        LLVM_DEBUG(dbgs() << "no callee for " << CSI.DestId << " " << CSI.Count


Suggested change

LLVM_DEBUG(dbgs() << "no callee for " << CSI.DestId << " " << CSI.Count

LLVM_DEBUG(dbgs() << "BOLT-DEBUG: no callee for " << CSI.DestId << " " << CSI.Count

maksfb · 2025-11-18T04:48:53Z

bolt/lib/Profile/YAMLProfileReader.cpp

+      }
+      // Get callee GUID
+      if (Callee->InlineTree.empty()) {
+        LLVM_DEBUG(dbgs() << "no inline tree for " << Callee->Name << '\n');


Suggested change

LLVM_DEBUG(dbgs() << "no inline tree for " << Callee->Name << '\n');

LLVM_DEBUG(dbgs() << "BOLT-DEBUG: no inline tree for " << Callee->Name << '\n');

maksfb · 2025-11-18T04:49:20Z

bolt/lib/Profile/YAMLProfileReader.cpp

+  });
+
+  assert(!Root.isRoot());
+  LLVM_DEBUG(dbgs() << "matchInlineTreesImpl for " << BF << "@"


Suggested change

LLVM_DEBUG(dbgs() << "matchInlineTreesImpl for " << BF << "@"

LLVM_DEBUG(dbgs() << "BOLT-DEBUG: matchInlineTreesImpl for " << BF << "@"

maksfb · 2025-11-18T04:49:56Z

bolt/lib/Profile/YAMLProfileReader.cpp

+
+  // Match profile function with a lead node (top-level function or inlinee)
+  if (Root.Guid != FuncNode.GUID) {
+    LLVM_DEBUG(dbgs() << "Mismatching root GUID\n");


Suggested change

LLVM_DEBUG(dbgs() << "Mismatching root GUID\n");

LLVM_DEBUG(dbgs() << "BOLT-DEBUG: mismatching root GUID\n");

maksfb · 2025-11-18T04:50:17Z

bolt/lib/Profile/YAMLProfileReader.cpp

+    uint64_t BinaryHash = Decoder.getFuncDescForGUID(Root.Guid)->FuncHash;
+    uint64_t ProfileHash = FuncNode.Hash;
+    if (BinaryHash != ProfileHash) {
+      LLVM_DEBUG(dbgs() << "Mismatching hashes: "


Suggested change

LLVM_DEBUG(dbgs() << "Mismatching hashes: "

LLVM_DEBUG(dbgs() << "BOLT-DEBUG: mismatching hashes: "

and more cases below.

shawbyoung added 5 commits July 24, 2024 11:28

[𝘀𝗽𝗿] initial version

2ff3c68

Created using spr 1.3.4

Comment

56b45b1

Created using spr 1.3.4

Was accessing wrong YamlBF Hash, fixed

b851ca6

Created using spr 1.3.4

Changed ordering of matching

39ba717

Created using spr 1.3.4

Added check for YamlBF.Used in pseudo probe function matching

11af7f1

Created using spr 1.3.4

aaupov mentioned this pull request Sep 3, 2024

[BOLT] Add pseudo probe inline tree to YAML profile #107137

Merged

aaupov added a commit that referenced this pull request Sep 4, 2024

[BOLT] Add pseudo probe inline tree to YAML profile

66fe5d5

To be used for pseudo probe function matching (#100446). Test Plan: updated pseudoprobe-decoding-inline.test Pull Request: #107137

shawbyoung added 2 commits September 11, 2024 15:35

Less naive pseudo probe function matching: for each non-matching Yaml…

1c8b150

…BF, determine best matching binary inline subtrees, find corresponding binary functions, attach the profile to binary function without a profile and most probes Created using spr 1.3.4

Debug logging

ad4d98f

Created using spr 1.3.4

aaupov mentioned this pull request Nov 8, 2024

[BOLT] Match blocks with pseudo probes #99891

Merged

Pseudo probe function matchign

3bd1ec2

Created using spr 1.3.4

clang-format

15643b7

Created using spr 1.3.4

rebase+update

43b0577

Created using spr 1.3.4

This was referenced Oct 28, 2025

[BOLT] Move call probe information to CallSiteInfo #165490

Merged

[BOLT][NFC] Split up StaleProfileMatching::matchWeights #165492

Open

shawbyoung added 3 commits October 28, 2025 21:53

rebase

7b8783e

Created using spr 1.3.4

clang-format

57f46da

Created using spr 1.3.4

Avoid structured binding

5dd720e

Created using spr 1.3.4

aaupov marked this pull request as ready for review October 29, 2025 06:25

aaupov requested review from aaupov, ayermolo, maksfb, paschalis-mpeis, rafaelauler, yota9 and yozhu as code owners October 29, 2025 06:25

wlei-llvm reviewed Nov 17, 2025

View reviewed changes

maksfb reviewed Nov 18, 2025

View reviewed changes

		uint64_t BinaryHash = Decoder.getFuncDescForGUID(Root.Guid)->FuncHash;
		uint64_t ProfileHash = FuncNode.Hash;

	/// Return a top-level binary inline tree node for a given \p BF
	/// Return a top-level binary inline tree node for a given \p BF.

	LLVM_DEBUG(dbgs() << "Matched yaml block (bid = " << YamlBB.Index << ")"
	LLVM_DEBUG(dbgs() << "BOLT-DEBUG: matched yaml block (bid = " << YamlBB.Index << ")"

	LLVM_DEBUG(dbgs() << "no callee for " << CSI.DestId << " " << CSI.Count
	LLVM_DEBUG(dbgs() << "BOLT-DEBUG: no callee for " << CSI.DestId << " " << CSI.Count

	LLVM_DEBUG(dbgs() << "no inline tree for " << Callee->Name << '\n');
	LLVM_DEBUG(dbgs() << "BOLT-DEBUG: no inline tree for " << Callee->Name << '\n');

	LLVM_DEBUG(dbgs() << "matchInlineTreesImpl for " << BF << "@"
	LLVM_DEBUG(dbgs() << "BOLT-DEBUG: matchInlineTreesImpl for " << BF << "@"

	LLVM_DEBUG(dbgs() << "Mismatching root GUID\n");
	LLVM_DEBUG(dbgs() << "BOLT-DEBUG: mismatching root GUID\n");

	LLVM_DEBUG(dbgs() << "Mismatching hashes: "
	LLVM_DEBUG(dbgs() << "BOLT-DEBUG: mismatching hashes: "

[BOLT] Match functions with pseudo probes #100446

Are you sure you want to change the base?

[BOLT] Match functions with pseudo probes #100446

Uh oh!

Conversation

shawbyoung commented Jul 24, 2024 • edited by aaupov Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Extended function and block pseudo probe stale matching

Prerequisites

Performance testing

Uh oh!

github-actions bot commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shawbyoung commented Jul 24, 2024 •

edited by aaupov

Loading

github-actions bot commented Nov 8, 2024 •

edited

Loading